Mark Indicators

Mark Indicators¶

In [ ]:

# ignore datajoint+jupyter async warnings
import warnings

warnings.simplefilter("ignore", category=DeprecationWarning)
warnings.simplefilter("ignore", category=ResourceWarning)
warnings.simplefilter("ignore", category=UserWarning)

nwb_copy_file_name = "J1620210531_.nwb"

# ignore datajoint+jupyter async warnings import warnings warnings.simplefilter("ignore", category=DeprecationWarning) warnings.simplefilter("ignore", category=ResourceWarning) warnings.simplefilter("ignore", category=UserWarning) nwb_copy_file_name = "J1620210531_.nwb"

In order to use clusterless decoding, you need to populate the UnitMarksIndicator table. This involves populating a series of tables (SpikeSorting -> CuratedSpikeSorting -> UnitMarks -> UnitMarkIndicators)

This may seem confusing at first, as the point of clusterless decoding is to avoid spike sorting. However, in order to keep the pipeline for spike sorting and clusterless mark extraction similar as possible, we use some shared steps. However, the "spike sorting" step involves just a simple thresholding operation (sorter: clusterless_thresholder) and the curation step is essentially skipped (although the entry still needs to be populated). It also make sense to only extract the mark indicator functions time periods where there is position data.

In order to simplify some of these steps, there is a function called populate_mark_indicators. In order to use it, you must first populate SpikeSortingSelection and IntervalPositionInfo.

Note that SpikeSortingSelection itself depends on:

• SpikeSortingRecording
• SpikeSorterParameters
• ArtifactRemovedIntervalList.

and SpikeSortingRecording itself depends on:

• SortGroup
• SortInterval
• SpikeSortingPreprocessingParameters
• LabTeam

You will have to define/populate all these things before getting to the "spike sorting" step. In order to read up on these steps, it is best to refer to tutorial notebook 1_spikesorting.ipynb for the spikesorting related steps and 4_position_info.ipynb for populating IntervalPositionInfo.

For the purposes of this notebook, let's assume you've done these steps and now simply need to populate UnitMarkIndicators via SpikeSorting. Let's import the function and look at its docstring, which will tell us what inputs the function expects:

In [2]:

from spyglass.decoding.clusterless import populate_mark_indicators


?populate_mark_indicators

from spyglass.decoding.clusterless import populate_mark_indicators ?populate_mark_indicators

/home/edeno/miniconda3/envs/spyglass/lib/python3.8/site-packages/seaborn/rcmod.py:82: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  if LooseVersion(mpl.__version__) >= "3.0":
/home/edeno/miniconda3/envs/spyglass/lib/python3.8/site-packages/setuptools/_distutils/version.py:346: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
  other = LooseVersion(other)

Connecting edeno@lmf-db.cin.ucsf.edu:3306

Signature:
populate_mark_indicators(
    spikesorting_selection_keys: list,
    mark_param_name='default',
    position_info_param_name='default_decoding',
)
Docstring: <no docstring>
File:      /stelmo/edeno/nwb_datajoint/src/spyglass/decoding/clusterless.py
Type:      function

From the docstring, we see that the main thing the function expects is spikesorting_selection_keys. This is a list of dictionaries, where each dictionary specifies the primary keys that SpikeSorting requires to run. It is a list of dictionaries because we typically want to extract the marks for more than one electrode at a time. Here are the primary keys required by SpikeSorting:

In [3]:

from spyglass.spikesorting import SpikeSorting

SpikeSorting.primary_key

from spyglass.spikesorting import SpikeSorting SpikeSorting.primary_key

Out[3]:

['nwb_file_name',
 'sort_group_id',
 'sort_interval_name',
 'preproc_params_name',
 'team_name',
 'sorter',
 'sorter_params_name',
 'artifact_removed_interval_list_name']

Here is an example of what spikesorting_selection_keys should look like:

In [4]:

spikesorting_selections = [
    {
        "nwb_file_name": "J1620210531_.nwb",
        "sort_group_id": 0,
        "sort_interval_name": "raw data valid times no premaze no home",
        "preproc_params_name": "franklab_tetrode_hippocampus",
        "team_name": "JG_DG",
        "sorter": "clusterless_thresholder",
        "sorter_params_name": "clusterless_fixed",
        "artifact_removed_interval_list_name": "J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times",
    },
    {
        "nwb_file_name": "J1620210531_.nwb",
        "sort_group_id": 1,
        "sort_interval_name": "raw data valid times no premaze no home",
        "preproc_params_name": "franklab_tetrode_hippocampus",
        "team_name": "JG_DG",
        "sorter": "clusterless_thresholder",
        "sorter_params_name": "clusterless_fixed",
        "artifact_removed_interval_list_name": "J1620210531_.nwb_1_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times",
    },
]

spikesorting_selections = [ { "nwb_file_name": "J1620210531_.nwb", "sort_group_id": 0, "sort_interval_name": "raw data valid times no premaze no home", "preproc_params_name": "franklab_tetrode_hippocampus", "team_name": "JG_DG", "sorter": "clusterless_thresholder", "sorter_params_name": "clusterless_fixed", "artifact_removed_interval_list_name": "J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times", }, { "nwb_file_name": "J1620210531_.nwb", "sort_group_id": 1, "sort_interval_name": "raw data valid times no premaze no home", "preproc_params_name": "franklab_tetrode_hippocampus", "team_name": "JG_DG", "sorter": "clusterless_thresholder", "sorter_params_name": "clusterless_fixed", "artifact_removed_interval_list_name": "J1620210531_.nwb_1_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times", }, ]

WARNING: Remember that this will only work if you have populated both SpikeSortingSelection and IntervalPositionInfo. You can check this by running the following and making sure there are table entries:

In [5]:

from spyglass.spikesorting import SpikeSortingSelection

SpikeSortingSelection & spikesorting_selections

from spyglass.spikesorting import SpikeSortingSelection SpikeSortingSelection & spikesorting_selections

Out[5]:

Table for holding selection of recording and parameters for each spike sorting run

nwb_file_name name of the NWB file	sort_group_id identifier for a group of electrodes	sort_interval_name name for this interval	preproc_params_name	team_name	sorter	sorter_params_name	artifact_removed_interval_list_name	import_path optional path to previous curated sorting output
J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times
J1620210531_.nwb	1	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_1_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times

Total: 2

Remember to replace the nwb_file_name with your own nwb file name

In [6]:

from spyglass.common import IntervalPositionInfo

IntervalPositionInfo & {
    "nwb_file_name": "J1620210531_.nwb",
    "position_info_param_name": "default_decoding",
}

from spyglass.common import IntervalPositionInfo IntervalPositionInfo & { "nwb_file_name": "J1620210531_.nwb", "position_info_param_name": "default_decoding", }

Out[6]:

position_info_param_name name for this set of parameters	nwb_file_name name of the NWB file	interval_list_name descriptive name of this interval list	analysis_file_name name of the file	head_position_object_id	head_orientation_object_id	head_velocity_object_id
default_decoding	J1620210531_.nwb	pos 0 valid times	J1620210531_XOKZ4G53LE.nwb	200c54a9-57dc-4ed4-8e48-85cc9387b922	26f4ec71-2f5f-4ef2-82f1-ee3a89048168	7f9ffe2f-31e5-47ac-bad9-558440cd5f1d
default_decoding	J1620210531_.nwb	pos 1 valid times	J1620210531_B257TS35XJ.nwb	c164cdfd-c328-49b2-b6b4-f8663a17f5d6	e1f26617-c44b-4464-942d-5e6755606789	f3c69ac6-5f0a-4622-827b-44a769ce4f77
default_decoding	J1620210531_.nwb	pos 10 valid times	J1620210531_1RICS57YYG.nwb	2394ec2d-c076-4b11-a40b-91e4b51b81fa	005080ee-b93f-4b3c-bcd0-595b46fedab2	47158c4f-3632-4ce1-9f1e-043f1ea5a5ec
default_decoding	J1620210531_.nwb	pos 11 valid times	J1620210531_HZSI1BGTE7.nwb	295aa187-81cd-4209-b96d-163f12a716ed	19a95d85-8cb9-4e06-82ee-26bc6df7c8c8	9abb39ce-f0e2-4637-b9e9-5f7b6891201d
default_decoding	J1620210531_.nwb	pos 12 valid times	J1620210531_M0UMC7TZ49.nwb	ae4c015c-9d5f-4ca3-bbcd-049c1e1ae50b	1bbc244c-4d85-4e63-9d69-10815e37276c	be8f6ba5-3cc9-4ac3-96af-5c3e84b9851c
default_decoding	J1620210531_.nwb	pos 13 valid times	J1620210531_TREA1SOT03.nwb	c4bfe562-f523-495c-99e5-681524661264	65023b5e-321b-4d6e-b975-ec03cbd46247	ce391291-6107-420a-a73d-9a18d343a47a
default_decoding	J1620210531_.nwb	pos 14 valid times	J1620210531_93PXV7F4CX.nwb	8521f813-635d-41cd-8f10-b00f26a7f957	75d08124-57f7-481b-ae44-0752a5912c18	84ce4054-8c11-49f2-8295-7d8173cb7eb7
default_decoding	J1620210531_.nwb	pos 15 valid times	J1620210531_FUJ1B1OKOA.nwb	e72e5fcd-3d80-463a-a9d1-305e6e99f8db	6e8dc0e2-ab02-4305-bf2e-e2d208bae77a	b7589719-49ba-48e7-a20b-88a911c3f8a5
default_decoding	J1620210531_.nwb	pos 16 valid times	J1620210531_76HL0SNP59.nwb	b3124712-55db-4b0b-9bdc-77f03a0b8c98	08bfd82c-1e7c-4aa7-bd9e-256577b26e9c	85ffd435-3dc4-49ab-adea-a3327817a51d
default_decoding	J1620210531_.nwb	pos 17 valid times	J1620210531_5F8VRT0OC4.nwb	78b8f7e4-fb90-438d-8551-d15a4d97484a	7b96d15c-9721-44ea-9ecd-d8a2b60711d6	12336768-227e-4c0c-9ade-b3c7ac6c2de3
default_decoding	J1620210531_.nwb	pos 18 valid times	J1620210531_PP7XK1G26S.nwb	39ab3238-a0dd-4906-95c5-8e534458e371	ca0f939d-f0a7-4df4-9946-2f605ef5ab76	571277d0-885e-4d0e-b802-6ad8047ee3f1
default_decoding	J1620210531_.nwb	pos 19 valid times	J1620210531_AQZHSP2IDM.nwb	b5a21053-3239-4943-aee1-3227b4915d5f	9f0c61c8-90ed-4a74-873c-e250aa872a2a	714f6ceb-6d70-474c-8c42-6f9fa3ff77e1

...

Total: 21

Now that we've checked those, we can run the function:

In [7]:

populate_mark_indicators(spikesorting_selections)

populate_mark_indicators(spikesorting_selections)

We can verify that this worked by importing the UnitMarksIndicator table:

In [8]:

from spyglass.decoding import UnitMarksIndicator

UnitMarksIndicator & spikesorting_selections

from spyglass.decoding import UnitMarksIndicator UnitMarksIndicator & spikesorting_selections

Out[8]:

curation_id a number correponding to the index of this curation	nwb_file_name name of the NWB file	sort_group_id identifier for a group of electrodes	sort_interval_name name for this interval	preproc_params_name	team_name	sorter	sorter_params_name	artifact_removed_interval_list_name	mark_param_name a name for this set of parameters	interval_list_name descriptive name of this interval list	sampling_rate	analysis_file_name name of the file	marks_indicator_object_id
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 0 valid times	500.0	J1620210531_NS19JB65RU.nwb	d0de79cc-f2be-4e86-b4e8-e58a8aef1c02
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 1 valid times	500.0	J1620210531_YP9MXAPE2G.nwb	7598dc69-a024-4140-8b37-d1342a0e7829
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 10 valid times	500.0	J1620210531_AGOI7DXRQX.nwb	12a60beb-3ac2-41ae-a091-eebd0b14672c
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 11 valid times	500.0	J1620210531_VXGQCG8UYO.nwb	d0d40c96-e5a1-494f-b8a7-bf475304d640
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 12 valid times	500.0	J1620210531_NF8RWZGCSG.nwb	2b6476a7-143f-4760-bafa-e4959f0e735a
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 13 valid times	500.0	J1620210531_F7OHK7UMO3.nwb	84aea599-dbf1-4d11-bca2-00aa6ae9967a
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 14 valid times	500.0	J1620210531_MPXHIH43Y1.nwb	78224e49-0118-458d-9aee-727db98fc306
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 15 valid times	500.0	J1620210531_YFG1894358.nwb	721526f8-df6d-442e-84ee-bf7dd6ff8c19
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 16 valid times	500.0	J1620210531_VJ5V0LRJKU.nwb	39258f97-4508-4068-9b2b-ce419a5d718d
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 17 valid times	500.0	J1620210531_NACXLH30KV.nwb	bf312f88-af31-4247-83dc-9633d191401a
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 18 valid times	500.0	J1620210531_LFKSWYVVRS.nwb	af89e3e1-71f7-4c7c-a5ef-94d3cc7a1203
0	J1620210531_.nwb	0	raw data valid times no premaze no home	franklab_tetrode_hippocampus	JG_DG	clusterless_thresholder	clusterless_fixed	J1620210531_.nwb_0_raw data valid times no premaze no home_franklab_tetrode_hippocampus_JG_DG_group_0.8_2000_8_1_artifact_removed_valid_times	default	pos 19 valid times	500.0	J1620210531_R9V8JD4IQA.nwb	3616a7b7-fca2-43bb-9c72-44eb3c22af80

...

Total: 42